Event stream processing (ESP) systems are used to process, store, and retrieve data streams having that require high transactional throughput rates. ESP systems have typically focused on in-memory databases to achieve high performance, but there are several classes of ESP systems for which in-memory databases are not well suited. Some systems require, for example, the persistence of some subset of the events processed and/or results that are produced at very high data rates, a feature which in-memory databases cannot provide. A persistent storage device, such as a hard disk, may be used to provide these features, and, indeed, some other classes of systems, such as event pattern matching and situation detection systems, do make use of persistent storage. Most high-performance ESP systems, however, are unable to tolerate the latency and degradation of throughput that results when data is stored to disk. Consequently, conventional ESP systems use techniques such as chronicling, estimation, and/or load-shedding, to store short-lived data using in-memory databases or custom indices to boost performance.
Log-structured stores (LSS) may also be used to increase the throughput of an ESP system. A conventional storage system is typically organized into a read-optimized repository with a write-optimized journal as a front-end to absorb the high volume of incoming transactions (for long lived data), coupled with a batch update mechanism for writing to the read-optimized repository. An LSS, in contrast, uses a log file that is constructed from a write-optimized journal, and appends data to the tail of the log. Although LSSs were initially thought to be a promising solution for write-intensive applications such as ESP systems, performance is hindered by the need for a background garbage collector or “cleaner” to reclaim free space from the log file. As a result, performance of the system degrades as the ratio of live data to free space increases because more and more time is spent waiting for the cleaner to re-locate live data while reclaiming very little free space.